{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/kaggle/input/nlp-getting-started/sample_submission.csv\n",
      "/kaggle/input/nlp-getting-started/test.csv\n",
      "/kaggle/input/nlp-getting-started/train.csv\n"
     ]
    }
   ],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
    "# For example, here's several helpful packages to load in \n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "# Input data files are available in the \"../input/\" directory.\n",
    "# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "# Any results you write to the current directory are saved as output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Hello world!**\n",
    "\n",
    "In this tutorial, I will got through usage of SOTA transformers opensourced by HuggingFace team.\n",
    "We will be using BERT transformer model for this tutorial.\n",
    "You can check this link to understand more about HuggingFace transformers https://huggingface.co/transformers/pretrained_models.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following are the basic steps involved in using any transformer,\n",
    "\n",
    "### **For preprocessing** \n",
    "1. Tokenize the input data and other input details such as Attention Mask for BERT to not ignore the attention on padded sequences.\n",
    "2. Convert tokens to input ID sequences.\n",
    "3. Pad the IDs to a fixed length.\n",
    "\n",
    "### **For modelling**\n",
    "1. Load the model and feed in the input ID sequence (Do it batch wise suitably based on the memory available).\n",
    "2. Get the output of the last hidden layer\n",
    "    * Last hidden layer has the sequence representation embedding at 0th index, hence we address the output as last_hidden_layer[0].\n",
    "3. These embeddings can be used as the inputs for different machine learning or deep learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Using BERT Transformer**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting transformers\r\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/50/10/aeefced99c8a59d828a92cc11d213e2743212d3641c87c82d61b035a7d5c/transformers-2.3.0-py3-none-any.whl (447kB)\r\n",
      "\u001b[K     |████████████████████████████████| 450kB 2.8MB/s \r\n",
      "\u001b[?25hCollecting sacremoses\r\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1f/8e/ed5364a06a9ba720fddd9820155cc57300d28f5f43a6fd7b7e817177e642/sacremoses-0.0.35.tar.gz (859kB)\r\n",
      "\u001b[K     |████████████████████████████████| 860kB 8.9MB/s \r\n",
      "\u001b[?25hRequirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from transformers) (2.22.0)\r\n",
      "Requirement already satisfied: sentencepiece in /opt/conda/lib/python3.6/site-packages (from transformers) (0.1.83)\r\n",
      "Requirement already satisfied: boto3 in /opt/conda/lib/python3.6/site-packages (from transformers) (1.10.29)\r\n",
      "Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.6/site-packages (from transformers) (2019.11.1)\r\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.6/site-packages (from transformers) (4.39.0)\r\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.6/site-packages (from transformers) (1.17.4)\r\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from sacremoses->transformers) (1.13.0)\r\n",
      "Requirement already satisfied: click in /opt/conda/lib/python3.6/site-packages (from sacremoses->transformers) (7.0)\r\n",
      "Requirement already satisfied: joblib in /opt/conda/lib/python3.6/site-packages (from sacremoses->transformers) (0.14.0)\r\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->transformers) (1.24.2)\r\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->transformers) (2.8)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests->transformers) (2019.9.11)\r\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->transformers) (3.0.4)\r\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->transformers) (0.9.4)\r\n",
      "Requirement already satisfied: botocore<1.14.0,>=1.13.29 in /opt/conda/lib/python3.6/site-packages (from boto3->transformers) (1.13.29)\r\n",
      "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /opt/conda/lib/python3.6/site-packages (from boto3->transformers) (0.2.1)\r\n",
      "Requirement already satisfied: docutils<0.16,>=0.10 in /opt/conda/lib/python3.6/site-packages (from botocore<1.14.0,>=1.13.29->boto3->transformers) (0.15.2)\r\n",
      "Requirement already satisfied: python-dateutil<2.8.1,>=2.1; python_version >= \"2.7\" in /opt/conda/lib/python3.6/site-packages (from botocore<1.14.0,>=1.13.29->boto3->transformers) (2.8.0)\r\n",
      "Building wheels for collected packages: sacremoses\r\n",
      "  Building wheel for sacremoses (setup.py) ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \bdone\r\n",
      "\u001b[?25h  Created wheel for sacremoses: filename=sacremoses-0.0.35-cp36-none-any.whl size=883999 sha256=7af5c243ae1058fcbfdfaa1211a88a5eba27a2c7ce75b59ea26d8fc817b8b981\r\n",
      "  Stored in directory: /root/.cache/pip/wheels/63/2a/db/63e2909042c634ef551d0d9ac825b2b0b32dede4a6d87ddc94\r\n",
      "Successfully built sacremoses\r\n",
      "Installing collected packages: sacremoses, transformers\r\n",
      "Successfully installed sacremoses-0.0.35 transformers-2.3.0\r\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "from tqdm import tqdm\n",
    "\n",
    "from transformers import BertTokenizer, BertModel\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_TYPE = 'bert-base-uncased'\n",
    "MAX_SIZE = 150\n",
    "BATCH_SIZE = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('/kaggle/input/nlp-getting-started/train.csv')\n",
    "test_df = pd.read_csv('/kaggle/input/nlp-getting-started/test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Load the required tokenizer and model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained(MODEL_TYPE)\n",
    "model = BertModel.from_pretrained(MODEL_TYPE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Convert Text to Tokens**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_input = train_df['text'].apply((lambda x: tokenizer.encode(x, add_special_tokens=True)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[101, 3224, 2543, 2379, 2474, 6902, 3351, 21871, 2243, 1012, 2710, 102]\n",
      "Here 101 -> [CLS] and 102 -> [SEP]\n"
     ]
    }
   ],
   "source": [
    "print(tokenized_input[1])\n",
    "print(\"Here 101 -> [CLS] and 102 -> [SEP]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Here 101 -> [CLS] and 102 -> [SEP]**\n",
    "\n",
    "[CLS] token refers to the classification token. We need to take the embedding of the token from the output layer. It represents entire sequence embedding.\n",
    "\n",
    "[SEP] refers to end of the sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's pad the sequence to fixed length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "padded_tokenized_input = np.array([i + [0]*(MAX_SIZE-len(i)) for i in tokenized_input.values])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  101  2256 15616  2024  1996  3114  1997  2023  1001  8372  2089 16455\n",
      "  9641  2149  2035   102     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0]\n"
     ]
    }
   ],
   "source": [
    "print(padded_tokenized_input[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's tell BERT to ignore attention on padded inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "attention_masks  = np.where(padded_tokenized_input != 0, 1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0]\n"
     ]
    }
   ],
   "source": [
    "print(attention_masks[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_ids = torch.tensor(padded_tokenized_input)  \n",
    "attention_masks = torch.tensor(attention_masks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the sequence embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 39/39 [31:05<00:00, 47.84s/it]\n"
     ]
    }
   ],
   "source": [
    "all_train_embedding = []\n",
    "\n",
    "with torch.no_grad():\n",
    "  for i in tqdm(range(0,len(input_ids),200)):    \n",
    "    last_hidden_states = model(input_ids[i:min(i+200,len(train_df))], attention_mask = attention_masks[i:min(i+200,len(train_df))])[0][:,0,:].numpy()\n",
    "    all_train_embedding.append(last_hidden_states)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "unbatched_train = []\n",
    "for batch in all_train_embedding:\n",
    "    for seq in batch:\n",
    "        unbatched_train.append(seq)\n",
    "\n",
    "train_labels = train_df['target']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Now we have the train embeddings.This can be used as an input to other machine learning models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, Y_train, Y_test =  train_test_split(unbatched_train, train_labels, test_size=0.33, random_state=42, stratify=train_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References:  **\n",
    "\n",
    "http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/\n",
    "\n",
    "https://huggingface.co/transformers/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5100"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
